Communication Across Fault-Containment Firewalls on the SGI Origin

Scalability and reliability are inseparable in high-performance computing. Fault-isolation through hardware is a popular means of providing reliability. Unfortunately, such isolation also increases communication latencies: typically, one has to drop into and out of the kernel to communicate between failure domains. On the other hand, relaxing fault isolation domains allows e cient communication, but at the risk of failure propagation, and thus reduced reliability. We are concerned with nding a middle ground between these extremes. We rst review a few salient aspects of the SGI Origin2000 architecture, mentioning the hardware features germane to e cient communication, and building protectionrewalls. Then, we describe a mechanism for risk-free, point-to-point communication between processes on distinct failure domains. Quoting performance numbers, we show that the overheads of crossing domains render this mechanism unattractive for small messages. To address this issue, we describe a mechanism for controlled opening of the rewalls, thereby achieving explicit inter-partition shared-memory for communication. We describe the kernel software that addresses the resulting reliability issues, and discuss how familiar IPC mechanisms such as MPI and SysV shared-memory can use the explicit sharedmemory to advantage. Finally, based on the lessons learnt, we discuss some future directions, and draw concluding remarks.